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requirements for individual jobs into levels of proficiency* It has 
been developed as a multifunctional program with the interactive 
components of Job Profiling, Instructional Support, Reporting, and 
Assessment. The assessment component contains applied mathematics, 
applied technology, teamwork, and observation aspects. The American 
College Testing (ACT) program furnishes supplies for the Work Keys 
program and provides computer scoring. The system is a generally 
wel 1 ~devel oped and well-executed product. When used effectively it 
can assist companies with employee selection and aid students in the 
smooth transition from school to work. Work Keys assessments are not 
norm-referenced, but a norming sample was drawn using 14,584 
individuals to evaluate internal consistency. Research has also 
supported the validity of Work Keys. The ACT adheres to high 
standards in Work Keys and provides extensive client services, but 
budget restraints might hinder organizations in fully implementing 
the Work Keys System. Some limitations to the use of Work Keys are 
identified, but it is concluded that it provides individuals with 
career planning information regarding existing skill levels and 
levels needed to perform the jobs they want. (Contains 12 
references.) (SLD) 
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Title : Work Keys™ System 

System Components : Assessments, Job Profiling, Instructional Support, 

and Reporting. 

Publisher: American College Testing (ACT) 

Dates of Publication: The eight criterion- referenced skill assessments 

may be used in various combinations and their publication dates 
range from 1992 to 1995. 

Time required to administer: Varies with each assessment; ranges from 
35 to 80 minutes. 

Cost: The following assessment prices include Standard Score Reports, 
dipping and handling, and one Administrator's Manual for every 
25 examinees: Reading for Information, Locating Information, 

Applied Mathematics, and Applied Technology @ $4.00 per 
assessment; Teamwork and Observation @ $7.00 per assessment; and 
Writing and Listening @ $16.00 per set. Rapid scoring services 
and administration videotapes are offered at additional costs. The 
Instructional Support components (Targets for Instruction) are 
available for each assessment and vary from $12.00 to $40.00. Job 
Profiling services start at $4,000 for the first job, plus travel 
expenses for the job profiler(s); discounts are offered for 
contracts over $100,000 (ACT, 1994a). 

Brief Description of Purpose and Nature of Test 
Work Keys was developed in conjunction with a variety of 
consulting groups and provides a metric which translates skill 
requirements for individual jobs into levels of proficiency (McLarty and 
Palmer, 1994) . It has been developed as a multifunctional program with 
four interactive components. The Job Profiling component uses Work 
Keys' SkillPro software to systematically analyze jobs. During this 
procedure the Work Keys analyst meets with subject matter experts, 

(i.e., job incumbents and their supervisors) who select the tasks most 
important to their job and identify the skills required for effective 
performance on that job (McLarty and Palmer, 1994) . The Instructional 
Support component provides a series of guides to supplement existing 
curricula and training. The Reporting component includes Chart Essay 
Reports, Individual Reports, Roster Reports, and Vocational Information 
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Reports (ACT, 1994a) . The Assessment component enables individuals to 
demonstrate their competency in Work Keys skill areas by responding to 
situations, reading passages, mathematical problems, and messages 
similar to those found in a wide variety of jobs. Each assessment 
consists of four or five levels and each successive level is more 
complex than the previous level (McLarty and Palmer, 1994) . Each test 
has its own score scale with different starting levels (ACT, 1994b) . 

Applied Mathematics measures mathematical reasoning skills applied 
to work- related problems. Reading for Information measures skill in 
reading and understanding memos, bulletins, notices, letters, policy 
manuals, and government regulations. Locating Information measures 
skill in using information taken from workplace graphics such as 
diagrams, blueprints, floor plans, charts, and instrument gauges. 

Applied Technology measures skill in reasoning and applying basic 
technological principles of mechanics, electricity, fluid dynamics and 
thermodynamics. The Teamwork and Observation assessment prompts are 
videotaped scenarios. Teamwork measures the skill of choosing behaviors 
and actions which support and implement team goals. Observation 
measures skill in paying attention to instruction and noticing details 
in processes, safety procedures, and quality control standards. 

Listening and Writing are open response assessments administered via one 
audiotape containing the directions for transcribing messages from 
customers, co-workers, or suppliers. Each written response is scored 
twice, for details and accuracy and then for writing mechanics and style 
(ACT, 1995) . 
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Practical Evaluation 



The ACT furnishes postage-paid supplies for returning the 
materials for computer scoring and it is not uncommon for it to take 
several weeks to receive results (Deis, 1995) . The individual response 
booklet indicates the location for bubbling- in the multiple choice 
answers and writing the messages from the Listening/Writing assessments. 
The returned reports provide general descriptive information regarding 
how various groups scored and the multi -page individual report provides 
the skill level for each assessment, descriptions of the tasks 
associated with the skill level, some strategies for improving the skill 
level, and a summary sheet suitable for attaching to a resume (ACT, 
1994b) . The reader should understand that the resulting level score 
(i.e. levels 3 to 7) are not related to any other index of reading 
difficulty like grade level (Deis, 1995) . The information is designed 
to be used for course planning, career guidance, individual goal 
setting, and can be authorized for release to prospective employers. 

Work Keys can be given in large groups and the administrator must adhere 
to the procedures as specified in the manual (ACT, 1994b) . Because the 
Work Keys components are interactive, they can be used repeatedly to 
monitor an individual's skill levels and to document improvement in 
education and training programs (Scruggs, 1994). The Work Keys System 
is a generally well developed and well executed product and when used 
correctly can be an effective tool to assist companies with employee 
selection and development and to aid students in the smooth transition 
from school to work (Deis, 19 Z ) . 



Technical Evaluation 



Since Work Keys assessments are not norm- referenced; defining and 
selecting a sample of examinees to describe the technical qualities and 
developing a scale that would convey meaning in a clear manner to anyone 
concerned about the strengths and weaknesses of individuals, programs, 
schools, training programs was a challenging task. The norming sample's 
average age was 18 and was drawn from examinees from three Work Keys 
Charter States. Seventy- two percent indicated their ethnic origin as 
Caucasian and 53 percent of the 14,584 individuals were in 12th grade. 
The response- centered Gut tman- scaling procedure was used to 
simultaneously order examinees and items in the order of highest -to- 
lowest examinee score and of easiest- to-most difficult item. Based on 
pretest data, the Guttman scaling data indicated coefficients of 
reproducibility ranging from .88 to .90 and coefficients of scalability 
ranging from .56 to .61 (Vansickle, 1992). The selected- response 
assessments indicated internal consistencies (i.e., coefficient alphas) 
ranging from .75 to .85. The Work Keys assessments contain items at the 
lower levels that most individuals can answer correctly and items at the 
upper levels that only a few individuals can answer correctly. This 
configuration has the effect of lowering the coefficient alpha computed 
for the selected- response assessments (ACT, 1994b) . Coefficient alpha 
was computed for the ratings received for each prompt of the Listening 
and Writing assessments and internal consistencies ranged from .85 to 
.89. These calculations were based upon a total of 12 values; each 
prompt id scored twice and there are 6 prompts per assessment . 
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Items written and selected for the assessments are screened 
against EEOC guidelines to help ensure job- relatedness and fairness 
(McLartv and Palmer, 1994) . The tests were developed using a content 
validity strategy; the tests measure skills shown to be contained in the 
jobs. This content validation is directly connected to the detailed job 
analysis/ job profiling procedures which are based upon the same metric 
as the assessments. Criterion (actual job performance) validation 
studies have not been reported. Alternate- forms reliability studies and 
validity studies in business settings are in progress. As more jobs are 
profiled and examinees are hired, opportunities for establishing 
criterion validity of Work Keys will become available and should provide 
an index of how effectively the Work Keys tests predict job performance. 

The ACT has initiated several changes as of January, 1996, to 
improve the reliability of each assessment (J. R. McLarty, personal 
communication, January 10, 1996) . The Work Keys' previous method 
counted correct responses for each subset of items, where a subset of 
ir.ems was at a particular level on the scale. On the selected- response 
tests, the upgraded scoring will allow credit for a level based upon the 
cumulative correct responses of the total test items. For example, a 
score of 14 for a specific assessment might indicate 82% mastery of 
level 3, 62% mastery of level 4 and 42% mastery of level 5. The score 
conversion, based upon empirical data, will be set at the point or cut 
score, which corresponds to 80% mastery. The upgraded scoring for 
Listening and Writing will back off of the current exact score approach 
and give credit for a level with a score that is within one standard 
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error of the criterion score. The changes retain the content 
definitions which link to job profiles, the large item pool at each 
level, the sequential nature of item difficulty and maintain the 
criterion referenced score. It appears that the upgraded scoring will 
improve internal consistency reliability by about .10 for each 
assessment, allow for equating forms, and provide the ■underlying base 
for developing adaptive computer versions. The new materials and 
scoring supports became available January, 1996. 

Reviewer Comments 

As Work Keys is relatively new, comments from the field have been 
included along with comments from the one published test review by C. 
Scruggs (1994) . Information gathered about workplace skills from a Work 
Keys assessment meets many of the federal and state evaluation needs 
mandated by work-based learning initiatives, contract- training 
requirements, and skill-certification alternatives. There is continued 
need for empirical studies regarding validity and alternate- form and 
test -retest reliability. Although Work Keys has the potential to 
provide a system for teaching and assessing employability skills, the 
job profiling services cannot possibly cover all the jobs that industry 
might create (Scruggs, 1994) . 

In one validation study, analysis of the Federal Express Customer 
Service Agent job established applicant requirements at level 5 for 
Reading for Information and Listening and at level 3 for Writing; 
although these could not be considered absolute qualifiers (Hater, 

1992) . Participants in tue Drills Standards and Certification Project 
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(Carmichael, Lessem, and Johnson, 1995) indicated that the profiling 
process was a logical, appropriate way to develop an occupational skills 
inventory and helpful in curriculum revision. This project recommended 
an ordinal arrangement for distinguishing the skill levels (Carmichael 
et al., 1995). Work Keys' job profiling has assisted the Ohio 
Department of Education in identifying applied academic skills required 
for success in 34 occupational areas (Vocational Instructional Materials 
Laboratory, 1994) . 

Research by the ACT into the psychometric characteristics of the 
Listening and Writing tests which use the same tape-recorded messages as 
stimuli for written responses, indicated that this linkage introduces 
conceptual and statistical complications, including the possibility of 
correlated errors (Brennan, Gao, & Colton, 1995) . A Weber State 
University project observed that listening on the job required 
interaction and two-way communications, not a one way receipt of highly 
artifical messages as presented by Work Keys. This project recommended 
dropping the Listening assessment and giving the facts in written format 
so that the assessee would prepare a business communication, thereby 
improving the assessment of Writing (Deis, 1995) . 

The Weber State University project recommended Teamwork profiling 
video revisions to help prevent the tendency to inflate the profile 
levels, eliminating reading computer screens as a component of Locating 
Information, upgrading the Teamwork assessment video tape, providing a 
reasonably priced on-line and/or fax scoring process, the ACT conducting 
additional criterion validation studies, and the ACT regularly convening 
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a user conference to identify problems and gather ideas for improvement . 
This project found evidence of criterion validity for Reading for 
Information, Applied Technology, Teamwork, and Writing assessments and 
stated that overall the Work Keys System is well developed and can be an 
effective tool in facilitating the transitions from school to work 
(Deis, 1995) . 

Summary Evaluation 

There are both strengths and weaknesses associated with the Work 
Keys System. The ACT adheres to high standards and provides extensive 
user friendly client services; but budget restraints might hinder 
organizations in fully implementing the Work Keys System. It appears 
that the interpretation of profile levels is complicated by the variety 
of descriptors and lack of common discrete intervals for each skill 
area. The numerical scores, indicating level 3 (relatively simple) to 
level 7 (complex legal, technical, or regulatory) , might indicate to the 
casual reader that the skill level score refers to an academic grade 
level or to other reported levels such as those used by the National 
Adult Literacy Survey (National Center for Education Statistics, 1993) . 
Although the use of Work Keys for initial screening of job applicants 
and for program evaluations appear to be appropriate uses, the success 
or failure of the applicant or student seems to be directly related to 
their reading ability and general cognitive ability level. The 
excessive dependence on independently reading the assessment prompts may 
have a negative impact on seme of the examinees . 
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Work Keys provides individuals with career planning information 
regarding their existing skill levels and the skill levels needed to 
perform the jobs that they want. Employers can use the job skill 
information to help select or further train their employees; educators 
can use the job skill information to develop appropriate workplace 
curricula and instruction. The Work Keys components can be used 
repeatedly to monitor attainment of the generic workplace competencies, 
as specified, and to document improvement in education and training 
programs . 
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